Experiment Result: Two Phase Flow
Updates
- Debugged inference method
- With the updated inference method, did hyperparameter search, especially with learning rate, \(\lambda\)
- We were wondering why it seemed like there was not so much improvement.
- We retrained the model with updated eigenvector -> refer to Updated Experiment Setting
- Loss decreases faster, posterior
Previous Experiment Setting
- Dataset
- \(2000\) pairs of \(\{K, S^t(K)\}_{t=1}^8\).
- Train Test split: [1800, 200]
- FIM
- Number of observation = 10
- Number of eigenvector = 1
- For a single pair of datapoint, we obtain 1 FIM.
- Likelihood is difference between perturbed time series of \(\{S^t(K)\}_{t=1}^8\) with true time series of \(\{S^t(K)\}_{t=1}^8\).
- Hyperparameter
- Batchsize = 100
Updated Experiment Setting
- Dataset
- \(1000\) pairs of \(\{K, S^t(K)\}_{t=1}^8\).
- Train Test split: [800, 200]
- FIM
- Number of observation = 2
- Number of eigenvector = 1
- For a single pair of datapoint, we obtain 8 FIM as there are 8 different time steps.
- Likelihood is difference between perturbed single Saturation \(S^t(K)\) with true singe time step Saturation.
- Hyperparameter
- Batchsize = 100
Pipeline
- FNO-NF.jl: create two-phase flow dataset, eigenvector of FIM, and vJp
- Diff_MultiPhysics: train (written in pytorch) and posterior estimation
Training Result
To evaluate training result, we go over three things:
- Loss Behavior
- Forward Simulation
- Posterior Estimation
Loss/Learning Behavior
| Epochs | \(\lambda\) | Train Loss | Test Loss | |
|---|---|---|---|---|
| MSE/GM | MSE | |||
| FNO-MSE | 1000 | N.A. | \(3.3622 \times 10^{-8}\) | \(8.4016 \times 10^{-8}\) |
| FNO-PBI | 2000 | 150.0 | \(1.0436 \times 10^{-7}\) | \(1.044 \times 10^{-7}\) |
| FNO-PBI | 2000 | 1.0 | \(3.0028 \times 10^{-8}\) | \(8.0099 \times 10^{-8}\) |
| FNO-PBI | 1000 | 150.0 | \(2.6428 \times 10^{-7}\) | \(1.5976 \times 10^{-7}\) |
| FNO-PBI | 1000 | 20.0 | \(6.2106 \times 10^{-8}\) | \(9.2973 \times 10^{-8}\) |
| FNO-PBI | 1000 | 5.0 | \(6.3265 \times 10^{-8}\) | \(9.7524 \times 10^{-8}\) |
| FNO-PBI | 1000 | 1.0 | \(4.1154 \times 10^{-8}\) | \(8.6791 \times 10^{-8}\) |
| FNO-PBI | 1000 | 0.7 | \(4.2235 \times 10^{-8}\) | \(9.1102 \times 10^{-8}\) |
Updated loss table
| Epochs | \(\lambda\) | Train Loss | Test Loss | |
|---|---|---|---|---|
| MSE/GM | MSE | |||
| FNO-MSE | 1000 | 1.0 | \(6.5207 \times 10^{-8}\) | \(1.3088 \times 10^{-7}\) |
| FNO-PBI | 1000 | 1.0 | \(8.3925 \times 10^{-8}\) | \(1.3030\times 10^{-7}\) |
We now evaluate surrogate models in two different criteria, forward simulation and inverse problem.
Evaluation: Forward Simulation
Forward Simulation on test sample
Learned and True vjp (sanity check)
We observe that for MSE,
- Scale in the color bar does not match.
- The learned vjp looks noisy as there are some colors showing in the part where it should be just white.
But in PBI, we verify that the learned vjp and true vjp matches well.
- The scale of color bar matches correctly.
- The plot does not look noisy.
What other things can be evaluated in terms of forward simulation?
- Stability: predict longer saturation evolution 9th to 16th.
- Generalization: test with out of distribution test samples.
Toy example to test generalizability of MSE and PBI
Also, generated out of distribution samples:
Things to change in experiment for better forward prediction of PBI
But before conducting all those experiments, we might want to change the eigenvector. That is,
From single eigenvector for single datapair,\(\{K, S^t(K)\}_{t=1}^8\), we generate eight different eigenvector to better inform time dynamics of plume.
Currently, the number of observation is 10. Increase to 100.
Evaluation: Posterior Estimate
\[ min_{K} \|S_{\theta}(K) - S(K*) \|^2_2 \]
where: - K0 = H(K) - S_{}: Neural Network model
Choosing the best parameters for MLE optimization
We conduct hyperparameter search for the \(\lambda\). The number of epochs chosen were based on the loss plot convergence. If it converged, we stopped training.
This is unconstrained. When lambda = 1., it is not accurate.
Unconstrained
| Epochs | \(\lambda\) | Loss (MSE) | SSIM | |
|---|---|---|---|---|
| FNO-PBI | 800 | 1.0 | \(4.2823 \times 10^{-4}\) | \(63.8013\) |
| FNO-PBI | 400 | 20.0 | \(8.9021 \times 10^{-5}\) | \(55.5040\) |
| FNO-PBI | 300 | 50.0 | \(6.1867 \times 10^{-5}\) | \(55.5502\) |
| FNO-PBI | 200 | 100.0 | \(5.5757 \times 10^{-5}\) | \(57.0893\) |
| ————— | ——— | ——— | ———————— | —————- |
| FNO-MSE | 800 | 1.0 | \(3.0692 \times 10^{-4}\) | \(57.3091\) |
| FNO-MSE | 400 | 20.0 | \(5.3901 \times 10^{-5}\) | \(52.3515\) |
| FNO-MSE | 300 | 50.0 | \(3.4602 \times 10^{-5}\) | \(54.2848\) |
| FNO-MSE | 200 | 100.0 | \(2.9429 \times 10^{-5}\) | \(55.6442\) |
Updated Result
Originally, we wanted to use Normalizing Flow for our inference method. But because it takes quite a long time to train, for a quick evaluation, we first try least squares method. Out of all 200 test samples, I brought some interesting cases.
Conclusion
- As of right now, we don’t see significant difference between MSE and PBI model in terms of posterior estimate.
- It is likely undertrained.
Future Step
- TODO: Debug NS eigenvector and vjp.
- TODO: Want to generate the full dataset for Francis’ dataset (which might take 1 or 2 days).
- TODO: Try it on Jason’s dataset (Now that we fixed the problem with FIM computation, we are optimistic about the experiment, so we want to try it again.)
Question
- Do we want to train both models for a longer time?